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(54) interactive desktop system with adjustable image capture 



(57) An interactive desktop system comprising a 
work surface, a video projector for displaying images in 
a display area, an camera for capturing innages present 
on the work surface, and an image processing computer 
coupled to the projector and to the camera, and adjust- 
ment means for adjusting, in response to user action, 
the field ot view of the camera. In the preferred embod- 
iment, the adjustment means comprises a frame includ- 
ing a housing in which the camera is disposed, the frame 
being nrKJvable (in a nnanner similar to that of a shade of 



a desk lamp) whereby the field of view of the camera is 
altered. In another emtxxiiment, the image processing 
computer is adapted for modifying, in response to a user 
input, the dimensions of the display area and/or the res- 
olution of the images displayed therein. The image 
processing computer may include means, responsive to 
a user input designating a position on the work surface, 
for causing movement of the housing whereby the said 
designated position is moved to within the field of view 
of the camera. 
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Description 

This invention relates to a docunnent processing 
system in which a user manipuiates docunnents interac- 
tively via using a camera positioned above a desk, and 
more particularly to such a system having adjustable im- 
age capture. 

It is known from EP-A-495 622 to use a camera- 
projector arrangement positioned above a desk, in order 
to enable a user to select functions to be performed by 
selecting items located within the field of view of the 
camera. A video camera or scanner Is used to capture 
innages of documents on a desk, and feedback Informa- 
tion is displayed by means of a projection display. The 
functions include calculating and translating operations 
carried out on data (e.g., in a paper document) located 
on the desk. 

EP-A-622,722 discloses a system for generating 
new documents from originals containing text and/or im- 
ages emp toying e.g. a camera-projector system fo- 
cused on a work surface, in conjunction with a copier or 
printer. In use. the camera captures various manual op- 
erations carried out by the user, e.g. by pointing with 
fingers and tapping on the surface on the text or images 
in an original paper document on the surface and rep- 
resenting manipulatbns of the text or images. Feedback 
to the user is provkied by projection of an image onto 
the surface or onto the original, or using some other vis- 
ual display. 

A problem with the system of EP-A-622,722 is that 
its use of a camera and projector which are fixedly 
mounted on office furniture, bookshelves or the ceiling 
is that it is very bulky, difficult to set up, and difficult to 
move. For practical purposes, many potential users 
would find it impossible to set up over their existing 
desks. Another problem is the limited field of view of the 
camera and the fact that it is inconvenient to zoom it and 
point it at different locations on the desk. 

A further problem with the system of EP-A-622,722 
is that the projector has a limited resolutbn and a limited 
range, and there is a trade-off between the two. Some 
applications (.e.g. fine sketching or handwriting) require 
high resolution per square inch, while other applications 
(e.g. Architectural visualisation) require a large surface 
area. Any given projection must sacrifice one for the oth- 
er. One aspect of this invention concems an interaction 
technique and input device that addresses the area/res- 
olution trade-off. 

A further problem arises through the low resolution 
of the images produced by the video cameras of the 
above-described systems (which are based on current 
standard video technotogy) compared with that of print- 
ers and scanners. This leads to low accuracy when rec- 
ognising the text of selections, and poor quality when 
printing selections. When the user is working with rela- 
tively small selections at a time (words or drawings less 
than a few inches across), then one approach to solving 
this problem is to zoom the camera to focus only on the 



area of interest. The problem then is how to control the 
camera. One approach is to have the user manually ad- 
just the camera zoom before making a selection, but this 
requires an extra step, extra controls, and a monitor of 

5 some sort so that the user can see "through the cam- 
era". It woukj be desirable to have a system that could 
automatically control the camera to zoom in as tightly 
as possible on the area selected by the user. 

The present invention provides interactive desktop 

10 system comprising a work surface, a display device for 
displaying images in a display area, an image capture 
device for capturing images present on the work sur- 
face, and processor means, coupled to the display de- 
vice and saki capturing means, and adjustment means 

fS for adjusting, in response to user action, the field of view 
of the image capture device. 

The present invention further provides an interac- 
tive desktop system comprising a work surface, a dis- 
play device for displaying images, an Image capture de- 

20 vrce for capturing images present on the work surface, 
and processor means, coupled to the display device and 
said capturing means, a frame including a housing in 
which the image capture device is disposed, the frame 
being movable whereby the fieW of view of the image 

2S capture device is altered. 

An advantage of this system is its compatibility with 
existing office equipment, and provision of a very natural 
Interaction technique for controlling the zoom factor and 
where the camera and projector point (discussed further 

30 betow in section B). 

The present invention further provides an interac- 
tive desktop system comprising a work surface, a dis- 
play device for displaying images in a display area, an 
image capture device for capturing images present on 

3S the work surface, and processor means, coupled to the 
display device and said capturing means, the processor 
means including means for modifying, in response to a 
user input, the dimensions of the display area and the 
resolution of the images displayed therein. 

40 With existing technology, if the zooming and posi- 
tion of the projector are controllable at all by the user, 
the controls are independent from the size and view of 
the image projected. This forces the user to select a 
trade-off between size and resolution which once made, 

45 is difficult to change. The technique according to the in- 
ventton makes it easy to move snnoothly between all 
possible values of this trade-off. so the limitations of any 
particular position are less of a problem. This technque 
could be compared to the techniques of navigating in a 

50 virtual world (e.g. flight simulation) but is the opposite in 
a way. Instead of showing a different view, according to 
the user's position in the computer's virtual world, what 
is shown is a different view according to the computer's 
position in the user's real world. 

55 The present invention further provides an interac- 
tive desktop system comprising a work surface, a dis- 
play device for displaying images, an image capture de- 
vice for capturing images present on the work surface. 
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and processor means, coupled to the display device and 
said capturing means, a frame including a housing in 
which the image capture device is disposed, the frame 
being movable, the processing means including means, 
responsive to a user input designating a position on the s 
work surface, for causing movement of said housing 
whereby the said designated position Is moved to within 
the field of view of said image capture device. 

Preferably, the processing means includes: means 
for determining from a captured image whether the pre- 
determined form Is horizontally centred within the field 
of view of said image capture device, and if not, for caus- 
ing the innage capture device to pan until the predeter- 
mined form Is horizontally centred; and/or means for de- 
termining from a captured image whether the predeter- 
mined form is vertically centred within the field of view 
of said Image capture device, and if not, for causing the 
image capture device to tilt until the predetermined form 
Is vertically centred. 

Preferably, the processing means includes: means 
for determining from a captured image whether the pre- 
determined form is fully visible within the field of view of 
said Innage capture device, and If so, for causing the Im- 
age capture device to electronically zoom in until the 
predetenmined form is no longer fully visible; and means 
for determining, after the predetermined form is no long- 
er fully visible, from a captured image whether the pre- 
determined form is fully visible within the field of view of 
said image capture device, and rf not, for causing the 
Image capture device to electronically zoom out until the 
predetermined form is just fully visible. 

The predetermined form may comprise a selection 
rectangle shaped by a user, and said elements comprise 
the side of said rectangle. 

This technique has the advantage that a user can 
simply select an area on the work surface and the Image 
capture device (camera) is automatically panned, 
zoomed, and/or tilted into position so that the selected 
area is just within its field of view — enhancing resolu- 
tion of the captured image in the selected area. 

The display device preferably comprises a video 
projector. Alternatively, the display device may comprise 
a desk-nnounted VDU, such as a CRT monitor, or a flat 
panel display, such as a LCD display Preferably, the Im- 
age capture device comprises a video camera. 

Embodiments of the invention will now be de- 
scribed, by way of example, with reference to the ac- 
companying drawings, In which: 

Figure 1 shows a view of the overall system accord- 
ing to one embodiment of the invention; 
Figure 2 is a schematic block diagram of the com- 
puter used In the system of Fig. 1 ; 
Figure 3 illustrates an alternative embodiment em- 
ploying mirror-based projection in accordance with 
the invention; and 

Figure 4 shows a further embodiment of the inven- 
tion employing a lamp housing movable in three di- 



mensions, for projecting a display of variable size 
and resolution; 

Figures 5 shows a further embodiment, similar to 
that of Fig. 4, but maintaining the projected display 
in a different orientation; 

Figure 6 Is a flow chart illustrating the processing 
steps used in the systems of Figs 4 and 5; 
Figure 7 is a view of the work surface Illustrating a 
further embodiment of the invention accomplishing 
automatic pan/lil/zoom of the camera to user-se- 
lected areas on the work surface; and 
Figure 8 Is a flow chart illustrating the processing 
steps used In the system of Fig. 7. 

A. Basic System Configuration 

The general system configuration according to the 
invention, employing a video camera and projector po- 
sitioned above and focused on a work surface 18, is il- 
lustrated in Fig. 1 . The system 1 takes the general form 
of a conventional desk lamp, with the camera 22 and 
video projector 9 (comprising bulb 5 and LCD matrix 7) 
being mounted within lamp housing 3. The base housing 
2, which is suitably larger than that for a conventional 
lamp, provides a housing for computer hardware, for ex- 
ample having a standard microcomputer (PC) architec- 
ture. (Alternatively, the processing hardware may be 
packaged in a standard desktop PC housing, with the 
base housing 2 being simitar to a standard desk lamp, 
and the camera 22 and projector 9 being coupled to ap- 
propriate ports of the desktop PC by suitable cables (not 
shown).) It will be appreciated by persons skilled In the 
art that the invention may be implemented using a com- 
puter running Windows™ and equipped with a frame 
grabber board and appropriate interfacing circuitry (see, 
for example, Jahne B., Digital Image Processing, Ap- 
pendix B, Springer-Verlag, Heidelberg, 1991). 

Alternatively, the hardware configuration described 
in detail In published European application EP-A- 
622,722, may be emp toyed except that only interfacing 
circuitry Is provided In housing 2, which is in turn con- 
nected to the remote processing system. The camera 
22 and video projector 9 are then connected to Image 
processing hardware described in detail in that applica- 
tion. 

In another alternative, the housing may contain only 
the camera 22, with the system being implemented as 
described in commonly-owned GB patent application 
No. 9614694.9 (Agent's ref: R/96007), filed concurrently 
herewith. 

In Implementing the present invention, image 
processing techniques described In EP-A-622,722 may 
be used, modified, as appropriate and/or as described 
below. 

The computer is coupled to a network cable 16, 
which in conjunction with a conventional internal driver 
card (not shown) and appropriate software (such as Net- 
ware, available from Novell Corp.) enable the computer 
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to form part of a local (LAN) or wide area network (WAN), 
as is well known in the art. 

The lamp housing 3 Is suitably arranged to be mov- 
able in three dimensrans. exactly in the manner ol a con- 
ventional angle-poise lamp, enabling the camera 22 and 
projector 9 to be moved towards and away from the desk 
surface 18 as desired for particular operations (in Fig. 1 
the angle that the line of sight of the camera/projector 
makes with the vertical is exaggerated, for the purpose 
of illustration). As shown, a real paper document 24 is 
within the field of view of the camera 22, and an elec- 
tronic document 25 Is projected by video projector 9 on 
to the surface 18. At the top of the lamp housing 3 is 
further provided a rotary zoom control 11 , similar to that 
provided on conventional SLR and video cameras, but 
electronically linked to the computer processor so as to 
enable the user to perform control of the zoom setting 
of the camera 22. 

The hardware of the computer system 1 1s schemat- 
ically shown in Fig. 2. The configuration Is well known 
to persons skilled in the art and, for example. Is de- 
scribed in detail in The Art of Electronics, 2nd Edn, Ch. 
1 0, P. Horowitz and W. Hill, Cambridge University Press, 
1989. Stated briefly, the system comprises, connected 
to common bus 30, a central processing unit 32, mem- 
ory devices Including random access memory (RAM) 
34, read only menrtory (ROM) 36 and disk, tape or CD- 
ROM drives 38, keyboard 12 (not shown), riKJUse 14 
(not shown), printing, plotting or scanning devices 40, 
and A/D. D/A devices 42 and digital Input/output devices 
44 providing interfacing to external devices 46 such as 
camera 22, video projector 9 and the rest of the LAN 
(not shown). 

In use, the video camera 22 captures Images of the 
desk-surface 1 8 and these Images are displayed in real- 
time by the video projector 9 on the desk surface 1 8. 

The system must maintain a nnapping between the 
pixels in the camera's field of view and the pixels of the 
display. This mapping will change as the angle of the 
lamp housing 3 with respect to the desk changes, so 
either this angle must be kept fixed, or a quk^k self-cal- 
ibration procedure (see, for example, EP-A-622,722, or 
GB patent appltcatbn 95 210 72.0) can be used. 

A further embodiment of the invention Is Illustrated 
in Fig. 3(a). This embodiment is the same as the first- 
mentioned embodiment, except as described below. 
The system addresses a problem encountered with a 
system employing a camera 22 to scan documents lying 
on the desk 52, and a video projector 9 to display feed- 
back information onto the desk surface 18, i.e. the inte- 
gration of the video projector 9, which is necessarily 
bulky due to the size of Its LCD array projection light 
source and cooling fans, with the camera 22 that scans 
the desk. 

The solution is to attach a mirror 50 to the underside 
of the camera housing, and to mount the projector 9 sep- 
arately, pointing upwards at the mirror 50. Here, the pro- 
jector 9 may take the form of a standard overhead pro- 



jector having a lens 54 for use in adjusting focus; and 
the LCD panel 7 Is disposed on the upper surface of the 
projector. The angle of inclinatbn of the projector is ex- 
aggerated. As a result the image (having width P) is pro- 

5 jected back down onto the same region (having width 
F) of the desk as is scanned by the camera. The projec- 
tor may be positioned behind the desk, or under the desk 
projecting up through a hole, as shown in Fig, 3(a). The 
mirror 50 is hinged at A and a screw adjustment at B 

10 allows the angle of the mirror 50 to be adjusted so that 
the image Is projected on the appropriate part of the 
desk surface 1 8. 

The image must be reversed to counter the effects 
of reflection by a single mirror 50. Most projectors and 

IS LCD panels having this reversal capability built in. Also, 
steps may need to be taken to minimise the keystoning 
(foreshortening) distortion and focusing problems 
caused by the innage being projected at an angle to the 
vertical. Again, some self-contained projectors include 

20 the means to displace the lens with respect to the LCD 
array 7 that generates the image. This may be sufficient 
to eradicate keystoning altogether, as shown in Fig. 3 
(b). and at the same time to bring the entire Image Into 
focus. A small amount of keystoning can be tolerated by 

^5 the user without difficulty. 

B. Proiecting a display of arbitrary size, position and 
resolutbn 

30 Figure 4 Illustrates a further embodiment of the in- 
ventbn: this is exactly the same as the first-mentioned 
embodiment, except as described below. 

In this embodiment, the lamp housing 3 Is prefera- 
bly restricted to being directed straight down (I.e. the an- 

35 gie of the optical axis of the camera 22 and projector 9 
to the vertical is zero: It is perpendicular to the desk sur- 
face 1 8). (In an embodiment where this angle is variable 
(see, e.g. Fig. 1 ) an Inverse projective transformation is 
required to produce an undistorted image). 

40 In this embodiment, the lamp base 2 Is configured 
such that the system (and therefore the lamp housing 
3) can rotate about an axis 60. 

Provision is made for sensing not only the vertical 
distance z from the lamp housing 3 to the desk surface 

45 18, but the horizontal position of the lamp housing 3 in 
two dimensions. For this purpose, shaft encoders 
(which output electrical signals Indicative of (change of) 
angular position) are provided at points C, D and E. It 
will be appreciated that alternative means may be used 

so for measuring the distance z: an ultrasonic transducer 
such as are employed in autofocus cameras may be 
used, disposed on the lamp housing 3. 

As seen In Fig. 4(b), which shows a plan view, using 
the swivel base 2 means that the display region 62 ro- 

55 tates as the system is swivelled. This is compensated 
for by applying the reverse rotation (-G) to the innage be- 
fore displaying it. 

(A further alternative embodiment is illustrated in 
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Fig. 5: this is the same as in Fig. 4, except that a more 
complex double linkage Is used which maintains the dis- 
play region 62 orthogonal to the desk). 

In the embodiments of Figs. 4 and 5, as the distance 
z from the desk surface 1 8 varies, the focus of the pro- s 
jector 9 must be nnaintained. In accordance with this as- 
pect of the invention, adjustment Is nnade automatically 
based on measured changes in z. 

Figure 6 is a flow chart of the processing steps in- 
volved In adjusting the displayed (projected) innage fol- 
lowing an alteration (change in x, y or z in the co-ordinate 
system of Figs. 4 and 5) of the position of the lamp hous- 
ing 3 by the user. In Fig, 6, allowance Is made for varying 
9 (Fig. 4), but for the embodiment of Fig. 5, 6 = 0. 

As can been seen in Fig. 6, initially (step s2), the 
angles at the shaft encoders are sensed. If a changes 
In the angle(s) is detected (step s4), computations are 
performed (based on well known geometric relations) to 
determine from the shaft encoder angles the projector's 
position (x, y, z, (and 8)) in the 3-D space. If it is deter- 
mined (step s8) that a change in vertical position (z) has 
taken place, further cateulations are performed, based 
n the new value of z and the relevant geometric rela- 
tions, to compute (step s1 0) the size of the new display 
region. The focus for the projector Is then adjusted (step 
s12) by a corresponding amount to ensure a sharp Im- 
age. Next, at step s14, the position of the display region 
in the virtual display is computed, and then a rotation of 
-6 is then performed the counteract any detected rota- 
tion 6. Then (step s18), operatbns are performed to 
render the display region onto the full region of the dis- 
play, and the rendered image displayed (step s20). 

The system maintains a virtual projected surface 
that Is of arbitrarily large size and arbitrarily high reso- 
lut bn. At any given moment the projector is displaying 
a portk>n of this surface at a particular resolutk>n. The 
user is given the impression of arbitrary size and reso- 
lutbn because he or she can move the projector (or 
zoom it) ck>ser to the desk and virtual objects stay the 
same size but are displayed with more pixels per inch. 
The size of the area can be made bigger by zooming (or 
moving) away, or the projector can be moved to the area 
that needs to be seen. This system is ideally suited to 
the system of Figs. 1 ,4 and 5, where the user can "light 
up' any part of the virtual surface area desired by mov- 
ing the lamp around. Then any area can be inspected 
to arbitrarily high resolution by moving the lamp closer. 
It Is believed that this interaction technique would be fa- 
miliar and very easy to learn. The only hardware re- 
quired in addition is the sensors so the system knows 
how far it and its angle is from the surface it is projecting 
on. 

C. Automatic pan/zoom to selections on the desktop. 

A further embodiment of the Invention will now be 
discussed with reference to Fig. 7: this is exactly the 
same as the previous embodiments, except as de- 



scribed below. According to this embodiment, the cam- 
era 22 is provided with an electronically controllable 
zoom, and the system provides automatic control of the 
camera's zoom by projecting a feedback rectangle 66 
that is also usable (detectable) by the system itself. The 
projected selectbn rectangle 66 is suitably of a prede- 
termined grey level enabling its position to be deter- 
mined. Alternatively, within the Individual skJes of the se- 
lection rectangle may be included coded information (e. 
g., glyph codes: see, for example, US-A-5,168,147; US- 
A-5,091,066; and US-A-5, 128.525). 

According to this technique, when the system can 
"see" the (whole of) the selection rectangle 66 that the 
user positions on the paper document 24 within the cam- 
era's field of view 70, then it dynamically zooms in the 
camera until just before it cani "see" it anymore. If the 
system cani "see" rectangle 66, then It zooms the cam- 
era out until it Is visible. 

Panning is done in a simitar way: when one of the 
skies of the rectangle 66 cannot be seen within the field 
of view 70, then the camera is panned (by electron b 
control of the camera's angular orientation) until the side 
can be seen. Preferably, only certain border regions of 
the fiekj of view 70, instead processed to determine the 
presence of the rectangle 66, thereby improving 
processing speed. 

Figure 8 is a flow chart illustrating the processing 
step in performing the above described panning and 
zoom operations in accordance with this aspect of the 
invention. 

Initially (step s32), inputs defining the user's adjust- 
ment of the (size/location of the) selection rectangle are 
received, and the accordingly-modified selection rec- 
tangle continuously displayed (step s34) as feedback. 
If rt is determined (step s36) that the user has finished 
selecting (adjusting), and the user chooses to 'copy' 
what is within the selection rectangle (step s38), the im- 
age (IMAGE1) currently within the camera field of view 
is grabbed (step s40). Then, the displayed (feedback) 
rectangle is removed (step s42) from the displayed im- 
age, and the image (IMAGE2) now within the camera 
field of view grabbed (step s44). An operation is then 
performed (step s46) to generate from 1MAGE1 and 
IMAQE2 a difference image. 

Using the difference image, the position of the se- 
lection rectangle within that image is searched for (step 
s48), using well known pattern recognition techniques. 
If the rectangle is not detected (step s50), the camera 
is zoomed out by a predetermined amount, and step s48 
repeated. This continuous in a stepwise fashion until the 
rectangle is detected at step s50. 

Then, an analysis is made (step s54) of whether the 
rectangle is centred (as between left and right) within 
the camera field of view, and if not, stepwise panning in 
the appropriate direction carried out (step s56) until it is 
detected (step s54) that the rectangle is centred. 

Then, an analysis is made (step s58) of whether the 
rectangle is centred (as between top and bottom) within 
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the camera field of view, and if not, stepwise panning in 
the appropriate direction carried out (step s60) until it is 
detected (step s58) that the rectangle is centred. 

Then, an analysis is made (step s62) of whether the 
rectangle Is fully visible within the camera field of view, 
and if so, stepwise zooming-in carried out (step s64) un- 
til it is detected (step s62) that the rectangle is no longer 
fully visible. 

Then, an analysis is made (steps66) of whether the 
rectangle is fully visible within the camera field of view, 
and if not, stepwise zooming-out carried out (step s68) 
until it is detected (step s66) that the rectangle is fully 
visible. Thus, the point is reached where the rectangle 
is now just fully visible. 



Claims 

1. An interactive desktop system comprising a work 
surface, a display device for displaying Images in a 
display area, an image capture device for capturing 
innages present on the work surface, and processor 
means, coupled to the display device and said cap- 
turing means, and adjustment means tor adjusting, 
in response to user action, the field of view of the 
image capture device. 

2. The system according to claim 1 , wherein the ad- 
justment means includes means for adjusting, in re- 
sponse to user action, a property of the displayed 
images. 

3. The system according to claim 1 or 2, wherein the 
adjustment means comprises a frame including a 
housing in which the image capture device Is dis- 
posed, the frame being movable whereby the field 
of view of the image capture device is altered. 

4. The system according to claim 3, wherein the ad- 
justment means includes a plurality of sensors for 
outputting a signal indicative of position; and where- 
in the processor means includes means for deter- 
mining the new position, following movement of the 
housing by a user, of said housing at least in two 
dimensions parallel to said work surface in depend- 
ence upon the signals output by said sensors. 

5. The system according to claim 3 or 4, wherein the 
frame is movable in a direction perpendicular to said 
work surface, the adjustment means includes a 
sensor for outputting a signal indicative of vertical 
position; and wherein the processor means in- 
cludes means for determining the new position, fol- 
lowing movement of the housing by a user, of said 
housing in three dimensions in dependence upon 
the signals output by said sensors. 

6. The system according to claim 3 or 4, wherein the 



frame is rotatable about an axis perpendicular to 
saki work surface, the adjustment means includes 
a sensor for outputting a signal indicative of angular 
position; and wherein the processor means in- 
5 eludes means for determining the new position, fol- 
bwing movement of the housing by a user, of said 
housing in three dimensions in dependence upon 
the signals output by said sensors. 

10 7. The system according to claim 4, 5 or 6, wherein 
the processor means includes means for rendering 
an altered image for display in dependence upon 
the determined new position. 

IS 8. The system according to any of the preceding 
claims, wherein said processor means includes 
means for modifying, in response to a user input, 
the dimensions of the display area and/or the reso- 
lutk^n of the images displayed therein. 

20 

9. The system according to any of claims 3 to 8, 
wherein the processing means includes means, re- 
sponsive to a user input designating a position on 
the work surface, for causing movement of said 

2S housing whereby the said designated position is 
moved to within the field of view of said image cap- 
ture device. 

10. The system according to claim 8, wherein the 
30 processing means includes means for determining 

from a captured image whether any of the elements 
of a predetermined form lie within the field of view 
of said image capture device, and if none of the el- 
ements are present, for causing the image capture 
35 device to electronically zoom out. 
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